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Method and arrangement for 
imaging and measuring 
microscopic three-dimensional 
a structures 

Ij Cross Reference to Related Applications 

hi 

Referenced-applicatlons 

^^^1 This invention claims priority of the German patent application 101 08 240.1 , filed on 

s February 21 , 2001 , which is incorporated by reference herein. 

W Background of Invention 

^ [0001] The invention concerns a method for imaging, measuring, and interacting with 
tU microscopic three-dimensional structures. The Invention further concerns an 

arrangement for imaging and measuring microscopic three-dimensional structures. 

[0002] As a user works at the microscope, image details (differing according to the 

application) are always in his or her field of view. In all microscopic applications, for 
example, there exists a class of functions for measuring morphological and 
densitometric properties. Morphology analyzes conformational and geometrical 
features, and densitometry analyzes intensity distribution features; in practice, mixed 
forms may often be encountered. This class of functions also constitutes the basis for 
many automatic adjustment operations. In present-day systems, the user works with 
image details by marking them on the screen with a suitable graphical mechanism 
(thus defining a geometry) and selecting a desired function of the system, the 
sequence described here being arbitrary. For three-dimensional geometries, this is a 
difficult process. 
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[0003] Three-dimensional geometric structures within three-dimensional image stacks 
that have been generated from data imaged with a microscope are difficult to 
apprehend mentally. In the real world, many users have only a partial three- 
dimensional conceptual capability, since 3D models demand a complex mental 
construction process and thus a certain amount of practice. This is particularly 
difficult when working with the raw data of a 3D microscope in the form of sectional 
images, in which case the user must completely reconstruct the three-dimensionality 
mentally. Measurement obliquely in space of continuous lengths that are not acquired 
in one section requires, on the basis of the acquired sections, complex navigation 
^.c^ through the stack of acquired two-dimensional image data. If the user has no ability 

^ to conceptualize in three dimensions, the structure of interest can be identified only 

cn 

with considerable effort. 

W 

'"'j [0004] Concrete examples from microscopy of actions in whose context this problem 
Ifj arises include: 

[0005] a)Statistical analysis of local properties of images and volumetric image stacks 
(profiles, histograms, co-localizations, material roughness); 

[0006] b)Observation of physiological reactions in living cells and in individual 

compartments (parts of a cell distinguishable in terms of metabolism or structure) 
thereof; 

[0007] c)Zoom operations; 
[0008] d)Aligning the image field; 
[0009] e)Controlling actuators; 

[001 0] f)Defining locally different excitation and detection parameters; 
[001 1] g)Automated control operations utilizing geometric data. 



[0012] 



The publication of D. Demandoix and J. Davoust, Multicolor Analysis and Local 
Image Correlation in Confocal Microscopy, Journal of Microscopy, Vol. 1 85 Pt. 1 , 
January 1 997, pp. 21-36 discloses a number of densitometric analysis methods. The 
individual analyses require both a geometric selection of the object to be analyzed 
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and geometric selections in a specific analytical space (the cytofluorogram). The 
publication of P. Wedekind. U. Kubitschek, and R. Peters, Scanning microphotolysis: A 
new photobleaching technique based on fast intensity modulation of a scanned laser 
beam and confocal imaging, Journal of Microscopy, Vol. 1 76, Pt. 1 , October 1 994, pp. 
23-33, discloses a capability for overlaying onto an acquired image of an object 
geometric elements that are differently illuminated on the specimen, and effecting 
changes in the specimen by way of the energy transport associated therewith. 

[001 3] Standard microscope systems make available usually implemented in software 
geometric models suitable for this purpose (polygons, rectangles, more generally a 
"region of interest" or ROI) that the user defines. This usually requires a time- 
consuming interactive process. The region is drawn with the mouse on the display of a 
computer system; then it is made available to the corresponding automation function. 

Summary of Invention 

[001 4] It is the object of this invention to create a method in which the measurement of 
three-dimensional structures is possible even without any spatial conceptual ability. In 
addition, the method according to the present invention makes it easy to define a 
subvolume that is to be imaged, 

[001 5] The stated object is achieved by way of a method that comprises the following 



steps: 



[0016] 



depicting a data set in three-dimensional form on a display associated with a 



microscope; 



[0017] 



defining at least one arbitrary section position and an arbitrary rotation angle; 



[0018] 



rotating the three-dimensional depiction on the display until a structure contained 
in the three-dimensional form reproduces on the display a depiction that appears 
suitable to the user for further processing; 



[0019] 



and performing an analytical operation on the structure. 



[0020] 



A further object of the invention is to create an arrangement in which the 
measurement of three-dimensional structures in an imaged specimen volume is 
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possible. The arrangement is intended to be independent of the user's three- 
dimensional conceptualization capability. In addition, specific scan parameters can be 
applied to arbitrarily selected subvolumes of the imaged specimen volume. 

[0021] The stated object is achieved byway of an arrangement which comprises: 

[0022] • a microscope, 

[0023] • a display associated with the microscope, 

[0024] • a PC, 

CI [0025] • an input device associated with the PC, 

"^^^ [0026] • a first window shown on the display for depicting a plurality of section planes, 

Hi 

id from viewing directions differing in each case, 



[0027] • a second window shown on the display for depicting a rotational view, 



r] [0028] • a third window shown on the display for a visual depiction of the coordinates, 



the rotation angle, and a section position; and 

[0029] • a respective crosshairs is shown in the depiction of each section plane, for 

interactive modification of the section position, wherein the image content in the first 
window is updated in accordance with a modification. 

[0030] The advantage of the invention is that the measurement of three-dimensional 

geometric structures within three-dimensional image stacks that have been acquired 
with a microscope is considerably simplified. The structure of three-dimensional 
objects is difficult to apprehend mentally, especially when working with raw data 
imaged by the microscope in the form of sectional images. In such a case the user 
must mentally reconstruct the three-dimensionality in order to stay oriented within 
the specimen. The method according to the present invention and the arrangement 
greatly facilitate this, by the fact that the requisite measurement means and 
adjustment devices, as well as a computationally ascertained reconstruction, are 
presented concurrently. This means less cognitive effort by the user and faster work, 
and thus more efficient utilization of the equipment (work vs. time). Measurement 
tasks, for example on lengths extending obliquely in the volume that are not acquired 
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in one section, are made substantially easier. If a structure of interest is not acquired 
in one section, then with conventional methods it must be discovered, in a time- 
consuming process, on the basis of the acquired sections by navigating through the 
image stack. This is made much more difficult if the user lacks three-dimensional 
conceptual ability, but is facilitated by the method according to the present invention 
and the arrangement. In the approach selected here, it becomes a simple matter of 
lining up and measuring. This combined utilization of minimal user inputs (mouse 
clicking) and the acquired microscopic image data to identify suitable section planes 
greatly enhances the ergonomics of the entire arrangement. In addition, the user is 
given a capability for easily navigating through the image stack without losing 
1 orientation. It is additionally advantageous that with the method, the data of three- 

dimensional subvolumes can be transferred directly to the microscope control system. 
The microscope control system can then apply appropriate or selected scan 
I parameters to the specific specimen volume. 

i 

Brief Description of Drawings 

[0031] The subject matter of the invention is depicted schematically in the drawings and 
will be described below with reference to the Figures, in which: 

[0032] FIG. 1 is a schematic depiction of a confocal scanning microscope in which the 
present invention is utilized. 

[0033] FIG. 2 shows an embodiment of a user interface for graphical depiction of the 
three different planes that are placed through a specimen volume. 

[0034] FIG. 3 is a schematic depiction of a cubic model for depicting the specimen 
volume and of an aid for the purpose of achieving orientation of the view upon 
rotation of the specimen volume. 

[0035] FIG. 4 shows a further embodiment of a user interface for determining a length in 
an imaged specimen volume. 

[0036] FIG. 5 shows a third embodiment for measuring areas and generating delimited 
volumes in the imaged specimen volume. 

Detailed Description 
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[0037] -| schematically shows a confocal scanning microscope. Light beam 3 coming 

from an illumination system 1 is reflected by a beam splitter 5 to scanning module 7, 
which contains a gimbal-mounted scanning mirror 9 that guides the beam through 
microscope optical system 1 3 over or through specimen 1 5. In the case of non- 
transparent specimens 1 5, light beam 3 is guided over the specimen surface. With 
biological specimens 1 5 (preparations) or transparent specimens, light beam 3 can 
also be guided through specimen 1 5. This means that different focal planes of 
specimen 1 5 are successively scanned by light beam 3. Subsequent assembly then 
yields a three-dimensional image of specimen 1 5. The image stack comprising a 
I plurality of two-dimensional images is assembled into a three-dimensional image. 

1 Light beam 3 coming from illumination system 1 is depicted as a solid line. Light 1 7 

] proceeding from specimen 1 5 comes through microscope optical system 1 3 and via 

I scanning module 7 to beam splitter 5, passes through the latter and strikes detector 

I 1 9, which is embodied as a photomultiplier. Light 1 7 proceeding from specimen 1 5 is 

depicted as a dashed line. Electrical detected signals 21 proportional to the power 
1 level of light 1 7 proceeding from specimen 1 5 are generated in detector 1 9 and 

= forwarded to processing unit 23. Position signals 25 sensed in scanning module 7 

with the aid of an inductively or capacitatively operating position sensor 1 1 are also 

I 

j transferred to processing unit 23, It is self-evident to one skilled in the art that the 

position of scanning mirror 9 can also be ascertained by way of the adjustment 
signals. The incoming analog signals are first digitized in processing unit 23. The 
signals are transferred to a PC 34 to which an input device 33 is connected. The user 
can, with reference to processing of the data, make corresponding selections by 
means of input device 33. In FIG. 1 , a mouse is depicted as an input device 33. It is 
self-evident to anyone skilled in the art, however, that a keyboard, joystick, voice 
input, and the like can also be used as input device 33. A display 27 depicts, for 
example, an image 35 of specimen 15. In addition, adjusting elements 29, 31 for the 
acquired image can also be depicted on display 27. In the embodiment shown here, 
adjusting elements 29, 31 are depicted as sliders. Any other configuration lies within 
the specialized ability of one skilled in the art. PC 34 forwards the corresponding data 
via line 37 to processing unit 23. The position signals and detected signals are 
assembled in processing unit 23 as a function of the particular settings selected, and 
are shown on display 27. The form in which the adjusting elements are depicted on 
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display 27 is immaterial for the invention. Illumination pinhole 39 and detection 
pinhole 41 that are usually provided in a confocal scanning microscope are 
schematically drawn in for the sake of completeness. Omitted in the interest of better 
clarity, however, are certain optical elements for guiding and shaping the light beams. 
These are sufficiently familiar to the person skilled in this art. 

[0038] The method is used for the imaging and structural measurement of microscopic 
three-dimensional structures in a specimen 1 5, the microscopic image having been 
acquired with a microscope. In this context, the three-dimensional structure can be 
arranged in any spatial position within specimen 1 5 and within the microscopic image. 
The microscopic image can be described as an acquired volume l(x) , where l(x) 
defines the vector of the acquired spectral components (channels or spectral image 
with the SP module) at volume position x of the voxel. A rotated specimen 1 5 or 
microscopic image can be generated using a rotation vector 

a = 



[0039] in which the individual angles (x and (x are the rotations about the 

X y 

corresponding axis in the mathematically positive direction (alternatively, and with no 
limitation as to generality, it is also possible to use YZ or XZ angle components, since 
the rotation of a body in three dimensions has only two degrees of freedom). Using 
the rotation matrix 



^1 0 0 ^ f cosct„ 0 sino: 



M(a)- 



0 coscE, - sin a, 
0 sin c, cos OL, 



■sina^ 0 cosa^j 



[0040] a rotated volume /(yj can be described in the coordinates y = M( a )x . This 
descriptive form is existing art in computer graphics (see Foley, van Dam, Feiner, 
Hughes: Computer Graphics Principle and Practice. 2nd ed. (1996) p. 215, Reading, 
Massachusetts, Addison Wesley). 

[0041] Utilizing the technique of "orthosectioning," arbitrary sections can be generated 
from this rotated volume. This is done by defining, in the rotated coordinates 

[0042] a position vector that implicitly defines three two-dimensional planes: 
plane 1 . 
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[0044] plane 2: 
[0045] plane 3: 

% : = O.z) y(t^) = (x = Ai,,>^,z) 

[0046] On the basis of the matrix relationship indicated above, these individual planes 
can be determined in the original volume using 

x(t^)=M(a)-V(t^) 

J [0047] The images along the oblique sections are of interest; these images can be 

} reconstructed by selecting various interpolation methods. Interpolated methods are 

1 established existing art, and the methods most familiar to those skilled in the art may 

I be looked up, for example, in Lehmann, G o nner, Spitzer: Interpolation Methods in 

1 Medical Image Processing. IEEE Transactions on Medical Imaging, Vol. 18, No. 1 1 

' (1999), pp. 1049-1075. All the methods listed therein are valid in connection with the 

1 method according to the present invention, and can be configured as options that 

I 

: result in different depiction quality levels. Interpolation methods of this kind are not 

I necessary in "orthosectioning," but are essential here. For interpolation, the values at 

J the plane positions are determined from adjacent voxels by linear weighting. This 

expanded form of "orthosectioning," and the three resulting images /{x ) , then 

ab 

make it possible to visualize any desired obliquely located sections, and provide 
access to any desired obliquely located linear geometric structures in the image. 

[0048] The method according to the present invention can be summarized as follows: In 
order to rotate the acquired three-dimensional image, the rotation angles are 
assigned. From the rotation angles, a transformation matrix is calculated. The desired 
section position within the three-dimensional image is determined, and then the 
planes are calculated. A back-transformation from the plane back into the original 
space is accomplished, and calculation of the sectional images is performed from the 
volume data by interpolation. The sectional images calculated in this fashion are then 
shown on display 27 as the result. 

[0049] It is evident to one skilled in the art that this method can be applied not only to 
the acquired raw data, but also to derived data (e.g. a ratio), an improved volume 
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(inverse filter), or a volume manipulated in anyway. 

[0050] As already mentioned above, a plurality of two-dimensional images are acquired 
by the microscope and ultimately assembled into the three-dimensional image stack. 
With the method according to the present invention it is also possible to navigate 
through the image stack. In order to adapt the method better to the user's sensory 
perception and allow it to be performed in brain-compatible fashion, the following is 
advisable: An arbitrary section position and arbitrary rotation angle are assigned; an 
initial setting can be allocated as the default setting. Calculation of the resulting 
section planes is performed in accordance with the method described above. The 
resulting section planes and the section geometry are depicted on display 27. 
Visualization is performed as a wire-frame model, or the acquired image can also be 
visualized three-dimensionally, in which case the rotation conforms to the angle of 
view and the sections in the three-dimensional volume are indicated. Also conceivable 
is a combination of the wire-frame model and three-dimensional visualization. For 
simplified adjustment of the rotation angle and section plane, adjustment elements 
are provided on display 27. Interactive modification of the angle and section plane 
parameters is performed by the user with simultaneous visual monitoring. The 
modification can continue until the structure being looked for is depicted on display 
27 in the form desired by the user. 

[0051] 2 shows one possible interface to be depicted for this purpose in display 27, 

with which the images and associated positional data are presented to the user. In the 

depiction on display 27, the "orthosectioning" view is combined with the view of a 

volume renderer and with a wire-frame object (see FIG. 3). FIG. 2 shows a preferred 

embodiment, the implementation being left to the specialized ability of one skilled in 

the art. On display 27, several individual elements are configured as virtual control 

elements. These individual elements are implemented by means of software in PC 34, 

and can be directly manipulated by the user by means of input device 33, e.g. a 

mouse, a voice input, or a touch screen. In the preferred embodiment, on display 27 

the standard "orthosectioning" view is depicted in a first window 40, supplemented by 

a rotational view in a second window 42. The "orthosectioning" view visualizes in first 

window 40 an image 40 of XY plane 41 , an image 40 of XZ plane 43, and an 

xy xz 

image 40 of YZ plane 45, images 40 ,40 , and 40 being lined up with one 
yz xy xz yz 
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another at the corresponding edges. The current position 

within images 40 ,40 , and 40 is indicated in each of the images by a 
xy xz yz 

crosshairs 70 ,70 , and 70 ; by manipulating crosshairs 70 .70 , and 
xy xz yz xy xz 

70 in the individual regions by means of a mouse or an equivalent input device, the 

yz 

section position 

can be interactively modified and the image content in the individual images 40 

xy 

40 , and 40 can be updated during the modification. A third window 44 on 
xz yz 

display 27 allows visual depiction of the coordinates, and shows both the rotation 
angle and the section position. The rotation visualization depicted in second window 
42 of the display ensures that the rotation can be modified as the user wishes. A wide 

variety of configurations is conceivable for this purpose. On the one hand, 

I 

J visualization could be accomplished by means of two nested cuboid models using line 

graphics, overlaid on which is an orthogonal three-dimensional crosshairs 70 , 70 

xy 

and 70 that reflects the position 
xz yz 

Xo = (^0 yo zq) 

1; 

( [0052] A depiction of the view of a wire-frame object using the cuboid models is shown 
in FIG. 3. An outer cuboid 60 represents the section geometry within whose 
boundaries the position y ^ can vary. Outer cuboid 60 also represents the view of a 
volume 62 to be examined. An arrow 63 symbolizes the transition to the depiction of 
the outer and inner cuboids, in which inner cuboid 62 is rotated and thus depicts the 
rotated volume. A corresponding software program links the rotation of inner cuboid 
62 to movements of the mouse or similar input devices. Inner cuboid 62 can be 
replaced by a volume reconstruction of the imaged specimen 1 5. This is done by 
using so-called volume rendering algorithms, which essentially simulate (more or less 
realistically, depending on the method) the way image creation proceeds 
macroscopically. This method makes it possible to create a realistic image of 
specimen 1 5, resulting a further decrease in the cognitive effort by the user. 

[0053] 

The actual configuration of these components is left to the artistic design 
capability of the implementing person skilled in the art. In addition to standard 
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visualization techniques such as average, maximum, or transparent projeaion, the 
simulated fluorescence projection (SFP) method designed specifically for fluorescence 
microscopy is also suitable, as well as all the enhancement and graphic design tricks 
of computer graphics. The depiction of the outer and inner cuboids 60 and 62 using 
line graphics and the depiction of inner cuboid 62 as a volume reconstruction of the 
imaged specimen 1 5 can be combined. 

[0054] The measurement of lengths in the imaged volume provides a further analytical 
capability. For determination of a length 50, a first and second position 51 and 52 in 
the imaged volume must be defined. The geometric distance between the two 
positions 51 and 52 is then the desired length 50. There are, in principle, two 
possibilities which allow a user to define positions 51 and 52 in the volume using the 
method. 

[0055] The user navigates into the plane in which the complete length 50 of interest is 
located, and marks the first and second positions 51 and 52. An input device 33, for 
example a mouse, panel box, voice input, etc. can be employed for this purpose. 
Another possibility is for the user to navigate into the plane in which the first position 

51 is located and mark it. He or she then navigates into the plane in which second 
position 52 is located, and marks it as well. Once the positions have been defined, PC 
34 can calculate and visualize the geometric distance. Navigation into the plane in 
which the structure to be measured is located is very much easier and more Intuitive 
for the user to do, and is already suitable for 99.9% of all measurement problems that 
occur in practice. 

[0056] 

FIG. 4 shows a depiction on display 27 that visualizes for the user the method for 
ascertaining a length 50. If the arrangement described in FIG. 2 is expanded to 
include control elements (not depicted) for drawing a line onto images 40 xz' 
and 40 , it is then possible to measure oblique lengths. The numerical lengths are 
displayed in a coordinate window 46 depicted on display 27. Before that, however, the 
volume must be aligned appropriately so that the user is suitably presented on display 
27 with the structure to be measured. Then the first position 51 and second position 

52 are determined. This can be done by direct marking of the points on display 27 or 
by displacing a ruler object (not depicted). PC 34 can determine therefrom the length 
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50 of the structure. The results are presented to the user, for example, in readable 
form in third window 44 on display 27. 

[0057] In the method for ascertaining a length in a volume, it is possible to make use of 
interpolation methods in order to utilize and visualize the intensity distribution (or 
similar parameters coded into the gray values). To do so, the volume is first aligned in 
accordance with the desired section plane as already described above. The length or 
lengths of interest in the volume can then be defined. Based on the defined lengths, 
the intensities are interpolated along them. The result is depicted in a chart. 
Interpolation is necessary because the discrete topology of the volume is not identical 
to the topology of the line, and must be adapted. Interpolation is the best possible 
way of estimating the distribution of the intensities on the line. The intensity graphs 
thus presented allow conclusions to be drawn as to specimen positions based on the 
quantitative profile of the intensity. More details are visible in the chart than in the 
image, which is subject to the limits of perception. 

[0058] In a further embodiment of the invention, obliquely oriented surfaces 54 are also 
measured. The properties of interest of the obliquely oriented surfaces in a volume 
can be determined by defining a region of interest (ROI). The properties are, for 
example, area, roundness, etc. An ROI in this context is a graphical figure that is 
overlaid on the image in order to describe the shape of the specimen in a manner 
desired by the user, and serves as a basis for analyses. As in the case of the other 
methods already described above, the plane must be appropriately aligned. The ROI is 
determined or selected using a corresponding graphical drawing tool or an automatic 
function. Surface 54 lying within a continuous line 56 is designated the ROI. On the 
basis of the selection, the size is calculated and then the result is presented to the 
user. Examples of automatic and semi-automatic functions for ROI determination are 
described, for example, in P. Hastreiter, T. ErtI: Fast and Interactive 3D Segmentation 
of Medical Volume Data, Proc, IMDSP 98 (1998), Alpbach, Australia ("intelligent 
cropping"); and in Blake, Isard: Active Contours, Heidelberg, Springer (1999) ("active 
contours"), and are to be regarded as existing art. Without the rotation and alignment 
components of the method described above, certain classes of ROIs are not 
accessible. 
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[0059] FIG. 5 shows an addition to or a further embodiment of the arrangement of the 

various windows on display 27. In FIG. 5, control elements 48 for drawing continuous 
line 56 onto the first window are added. 

[0060] A further embodiment of the method is to define multiple continuous lines 56 in 
different planes of the acquired image. Surfaces 54 resulting therefrom are assembled 
into a stack. The result is a three-dimensional volume on which certain analyses can 
be performed. This volume is directly accessible to measurement. Rotation of the 
plane is not essential here, but for certain volumes it does greatly simplify handling, 
since if the specimen is rotated appropriately, the continuous lines 56 of the sections 
j through the volume become simpler and the user does not need to draw complex 

I lines. Once the plane has been aligned using the method described above, a volume is 

I constructed from the basic surfaces. The desired sizes are then calculated and shown 

I on display 27. Once again, a considerable savings in working time is achieved. 

1 

I [0061] A further interesting use of the method is to create a capability which makes it 
^ easy to apply the scan parameters exclusively to a selected volume. Once the volume 

i has been determined in the manner described above, the scanning operation can, for 

I example, be applied exclusively to the specific sample. The determination of the best 

I possible scanning geometry for acquiring this selected structure or. volume is made by 

software, taking into account the microscope technique presently in use (e.g. 
objective, pinhole sizes, optical resolution) as well as signal-theory considerations 
(scanning theorem, Nyquist). With regard to microscope technique, the calculation 
takes into account what can technically be achieved with the type of microscope 
available and the equipment (such as objectives) that is available and installed. 
Applicable signal-theory considerations include which structure is resolvable and how 
often it needs to be scanned. Once the requisite parameters have been calculated, the 
scan is performed. 

[0062] 

As a further embodiment of the invention, arbitrary three-dimensional curves (not 
depicted) are defined within the acquired image. Scanning or bleaching then occurs 
along those curves. The three-dimensional curve is once again generated from a stack 
of multiple two-dimensional continuous lines 56. The scanning microscope is then 
controlled in such a way that only points on the three-dimensional spatial curve are 
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scanned. Scanning exclusively on the three-dimensional spatial curve makes possible 
not only data acquisition from that curve, but also modification of the specimen on 
the curve. The modifications of the specimen are, for example, bleaching or shifting. 

[0063] In addition, the method is used for Interactive determination of a point of interest 
in the specimen volume. The illumination parameters are then modified exclusively for 
the selected point. The point defines a small specimen volume, making possible 
transfer of a radiation dose exactly onto the specimen volume defined by the point. 
This can optionally be followed by a further scanning operation to observe recovery 
processes, or by the writing of further structures; once again, the second scanning 
operation acts exactly on the specimen volume. 
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